[WIP][v6r20] Space token usage#3601
Closed
chaen wants to merge 27 commits into
Closed
Conversation
…Usage * origin/spaceTokenUsage: (26 commits) Conflicts resolved Resolved conflicts v6r19p9 v6r19p9 notes testing on protocol using base types allow also space-separated arguments allow giving list of requests in a file using URLBase for everything but SRM2 case when an SE is at no sites added some doc added protection just style use resolveSEGroup avoid instantiating all local SEs marked getStorageElementOptions as deprecated style also considering CEs, not only SEs style docs and pylints ...
* upstream/integration: (25 commits) v6r19p10, v6r20-pre10 v6r19p10 v6r20-pre10 wrap lines better fix isinstance Added warning log message in case of exception log the delay only set delay once again logging fix logging delay execution if no active replicas found delay execution by 1 h if no active replicas found print NotBefore use defaultdict Catch exception when closing closed connection few more logging changes less logging in DM transfers limit logging fix case when no site is found for SE don't try and get SE metadata if replica is inactive ...
* upstream/integration: styled bug fix (double return) DMS: fix bad merge of ReplicateAndRegister
* upstream/integration: v6r19p11 v6r20-pre10 notes v6r19p11 notes using python modules always removing the sandboxes of deleted jobs just style just style added doc about shifterProxy 64 chars instead of 32 new option --Force to reset even non-failed requests + accept job IDs from a file new option --Maximum to limit the number of jobs selected + autopep8 autopep8 reset the NotBefore field for all requests when --Reset is used make pylint happier with data members RMS: add missing JOIN in the web summary query style fixed exception handling
…_SRM and DIP storages
Contributor
Author
|
__queryType is exists. I do not know why it is failing. |
fstagni
reviewed
Jan 22, 2018
| spaceTokenAttr = 'spacetoken.description?%s' % self.protocolParameters['SpaceToken'] | ||
| # gfal2 can take any srm url as a base. | ||
| spaceTokenEndpoint = self.getURLBase()['Value'] | ||
| print "%s %s" % (spaceTokenEndpoint, spaceTokenAttr) |
| occupancyStr = self.ctx.getxattr(spaceTokenEndpoint, spaceTokenAttr) | ||
| try: | ||
| occupancyDict = json.loads(occupancyStr) | ||
| except ValueError: |
Contributor
There was a problem hiding this comment.
Can you prepend the comment below with a FIXME?
|
|
||
| except (gfal2.GError, ValueError) as e: | ||
| errStr = 'Something went wrong while checking for spacetoken occupancy.' | ||
| self.log.verbose(errStr, e.message) |
Contributor
There was a problem hiding this comment.
Why a verbose message? Why not at least a warn?
| sTokenDict['Free'] = float(output.get('unusedsize', '0')) / 1e12 | ||
|
|
||
| sTokenDict['Total'] = float(occupancyDict.get('totalsize', '0')) / 1e6 | ||
| sTokenDict['Free'] = float(occupancyDict.get('unusedsize', '0')) / 1e6 |
Contributor
There was a problem hiding this comment.
I don't know if we use Guaranteed anywhere...? What was it for, do you know?
| "createDirectory": "createDirectory", | ||
| "putDirectory": "putDirectory", | ||
| "getDirectory": "getDirectory", | ||
| "getOccupancy": "getOccupancy" |
|
|
||
| It loops over the different Storage Plugins to query it. | ||
|
|
||
| :returns: S_OK with dict (keys: Total, Guaranteed, Free) |
Contributor
There was a problem hiding this comment.
Not the Guaranteed, it seems.
|
|
||
| log.debug("Negociating protocols between %s and %s (protocols %s)" % (sourceSE.name, self.name, protocols)) | ||
| log.debug( | ||
| "Negociating protocols between %s and %s (protocols %s)" % |
Contributor
|
There's a strange failure in travis (pylint), please just merge upstream/integration. |
Contributor
|
Actually integration fails: https://travis-ci.org/DIRACGrid/DIRAC/builds/330842302. I will have a look. |
Contributor
|
I merge integration in this branch so that Travis should succeed. |
Contributor
|
pylint in travis complains about can we add it to https://github.com/DIRACGrid/DIRAC/blob/integration/requirements.txt like we added fts3? |
Contributor
Author
|
We MAY try with https://gitlab.cern.ch/dmc/gfal2-bindings |
Contributor
|
In theory it should complain everywhere, but pylint isn't always spotting everything everywhere. |
Contributor
Author
|
We could do it, but it requires quite some compilation work, so I think we just ignore it with a pylint flag. I will do it now, and rebase |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR is meant to replace #3572
It contains a real implementation of the getOccupancy for the StorageElement, as well as an implementation for GFAL2_SRM2.
Note that the version of gfal2 we use at the moment has bugs (see comments) that will be fixed in 2.15. For the time being, I wrote workarounds.
BEGINRELEASENOTES
*Core
NEW: utility to convert file sizes between different unit
*Resources
NEW: Added StorageElement.getOccupancy() method for DIP and GFAL2_SMR2 SE types
*RSS
CHANGE: Using StorageElement.getOccupancy()
ENDRELEASENOTES